Reward modeling for mitigating toxicity in transformer-based language models

Transformer-based language models can generate fluent text and be efficiently adapted across various natural generation tasks. However, that are pretrained on large unlabeled web corpora have been shown to suffer from degenerating toxic content social bias behaviors, consequently hindering their safe deployment. Various detoxification methods proposed mitigate model toxicity; however, these struggle detoxify when conditioned prompts contain specific identities related gender, race, or religion. In this study, we propose Reinforce-Detoxify, a reinforcement learning-based method for mitigating toxicity in models. We address the challenge of safety new reward detect unintended towards prediction. The experiments demonstrate Reinforce-Detoxify outperforms existing approaches automatic evaluation metrics, indicating our approach is less prone toward generated content.